COUNTY OF 


ALLEGHENY 



Rich Fitzgerald 

County Executive 


May 3, 2019 


Beryl Lipton 
MuckRock News 
Dept. MR 70873 
411A Highland Avenue 
Somerville, MA 02144 

70873-26530511(S)requests.muckrock.com 

Re: RTK — Final Response 

Dear Ms. Lipton: 

This is in response to your attached request for records made pursuant the Pennsylvania Right- 
to-Know Law (RTKL). This Office received your request on March 25, 2019 and notified you on 
April 1, 2019 that an extension of time to respond to your request was necessary due to bona fide 
staffing limitations. 

Your request is granted in part and denied in part for the reasons set forth below. Each item of 
your request and the response to that specifically requested item is as follows. 

1. Your request is granted in part and denied in part. Section 704 (b)(1) of the RTKL provides that 
an agency may respond to a request by notifying the requestor that the record is available 
through publicly accessible electronic means. Records responsive to your request are publicly 
accessible at: 

https://www.alleghenvcountv.us/Human-Services/Resources/Doing- 
Business/Solicitations/Awarded-Contracts/Award-Details—023.aspx 

In addition, the following records, which have been redacted to remove information that is 
exempt from disclosure pursuant to RTKL Section 708(b)(6)(i)(A), which exempts "a record 
containing all or part of a person's Social Security number; driver's license number; personal 
financial information; home, cellular or personal telephone numbers; personal email 
addresses; employee number or other confidential personal identification number," are being 
provided on the enclosed CD: 


JerryTyskiewicz. Director 
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"Allegheny HealthChoices Proposal_Redacted.pdf' 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"Ciber Proposal_Redacted.pdf" 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"Community Human Services Proposal_Redacted.pdf." 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll) ; which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"Coordinated Care Services, Inc. Proposal_Redacted.pdf" 

"Data Warehouse Consultants Proposal_Redacted.pdf" 

"Deloitte Decision Support Proposal_Redacted.pdf" 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"Eckerd Proposal_Redacted.pdf" 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"KPMG Proposal_Redacted.pdf" 

"National Council on Crime and Delinquency Proposal l_Redacted.pdf" 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"National Council on Crime and Delinquency Proposal 2_Redacted.pdf' 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 
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"PCG Proposal_Redacted.pdf." 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"SAS Proposal_Redacted.pdf" 

"TCC Software Solutions Proposal_Redacted.pdf" 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"Tembo Proposal_Redacted.pdf" 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"Univ of Colorado Proposal_Redacted.pdf" 

2. Your request must be respectfully denied pursuant to RTKL Section 708(b)(26), which exempts 
"a proposal pertaining to agency procurement or disposal of supplies, services or construction 
prior to the award of the contract or prior to the opening and rejection of all bids; financial 
information of a bidder or offeror requested in an invitation for bid or request for proposals 
to demonstrate the bidder's or offeror's economic capability; or the identity of members, 
notes and other records of agency proposal evaluation committees established under 62 
Pa.C.S. § 513 (relating to competitive sealed proposals)." 

3. Your request is granted. Section 704 (b)(1) of the RTKL provides that an agency may respond 
to a request by notifying the requestor that the record is available through publicly accessible 
electronic means. Records responsive to this request are publicly accessible at: 

https://www.alleghenvcountvanalvtics.us/wp-content/uploads/2019/04/16-ACDHS- 
26 Ethical Package 102518.pdf#page=4 

https://www.alleghenvcountvanalvtics.us/wp-content/uploads/2018/10/17-ACDHS- 
11 AFST 102518.pdf 

https://voutu.be/A48e-p 3 Xs 

4. Your request is granted. The following records are being provided on the enclosed CD: 

"April community meeting notes.pdf" 

"Using Predictive Modeling to Improve Outcomes.pdf" 
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5. Your request is granted in part and denied in part. Section 704 (b)(1) of the RTKL provides that 
an agency may respond to a request by notifying the requestor that the record is available 
through publicly accessible electronic means. Records responsive to this request are publicly 
accessible at: 

https://www.alleghenvcountv.us/Human-Services/Resources/Doing- 
Business/Solicitations/Awarded-Contracts/Award-Details—005.aspx 

In addition, the following records, which have been redacted to remove information that is 
exempt from disclosure pursuant to RTKL Section 708(b)(6)(i)(A), which exempts "A record 
containing all or part of a person's Social Security number; driver's license number; personal 
financial information; home, cellular or personal telephone numbers; personal email 
addresses; employee number or other confidential personal identification number," are being 
provided on the enclosed CD: 

"Chapin Hall Proposal_Redacted.pdf" 

"Deloitte Predictive Risk Modeling Proposal.pdf" 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "A record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"PCG Proposal_Redacted.pdf" and "PCG Proposaljtedacted page 59.pdf 
This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"Pitt Proposal_Redacted.pdf" 

This record was redacted by the third party entity that created it to remove information that 
is exempt from disclosure pursuant to RTKL Section 708(b)(ll), which exempts "a record that 
constitutes or reveals a trade secret or confidential proprietary information." 

"Sarah Font Proposal_Redacted.pdf" 

6. Your request for the process evaluation is granted. Section 704 (b)(1) of the RTKL provides 
that an agency may respond to a request by notifying the requestor that the record is available 
through publicly accessible electronic means. The process evaluation is publicly accessible at: 

http://hornbvzeller.com/wp- 

content/uploads/2018/03/Predictive%20Analvtics%20Process%20Evaluation%20Alleghenv 

%20Countv.pdf#zoom=100 

Your request for the impact evaluation must be respectfully denied pursuant to RTKL Section 
708(b)(9), which exempts "the draft of a bill, resolution, regulation, statement of policy, 
management directive, ordinance or amendment thereto prepared by or for an agency." At 
this time the impact evaluation is not finished. 
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7. Your request is granted. Section 704 (b)(1) of the RTKL provides that an agency may respond 
to a request by notifying the requestor that the record is available through publicly accessible 
electronic means. Records responsive to this request are publicly accessible at: 

https://www.alleghenvcountvanalvtics.us/wp-content/uploads/2019/04/16-ACDHS- 
26 Ethical Package 102518.pdf#page=4 

Additionally, the following record is attached: 

"Table with both at the source levels.pdf" 

8. Your request is granted. Section 704 (b)(1) of the RTKL provides that an agency may respond 
to a request by notifying the requestor that the record is available through publicly accessible 
electronic means. Records responsive to this request are publicly accessible at: 

https://www.alleghenvcountvanalvtics.us/wp-content/uploads/2018/10/17-ACDHS- 
11 AFST 102518.pdf . 

9. Your request is granted. Section 704 (b)(1) of the RTKL provides that an agency may respond 
to a request by notifying the requestor that the record is available through publicly accessible 
electronic means. Records responsive to this request are publicly accessible at: 

https://www.alleghenycountvanalvtics.us/wp-content/uploads/2018/10/17-ACDHS- 
11 AFST 102518.pdf . 

10. Your request is granted in part and denied in part. Six documents are being provided on the 
enclosed CD: 

"Contracts - AUT ENTERPRISES LIMITED.pdf" 

"Contracts - AUT ENTERPRISES LIMITED - 7-1-2016 - 6-30-2018.pdf" 

"Contracts -BOARD OF TRUSTEES - 3-8-2016 - 3-31-2017.pdf" 

"Contracts -BOARD OF TRUSTEES - 3-8-216 - 7-31-2017.pdf" 

"Contracts - CARNEGIE MELLON UNIVERSITY - 11-15-2017 - 5-30-2020.pdf" 

"Contracts - HORNBY ZELLER.pdf" 

Allegheny County cannot ascertain how to search for additional records responsive to your 
request for "any correspondence and contract documents relating to the development and 
implementation of the program." Section 703 of the RTKL requires that requests "identify or 
describe the records sought with sufficient specificity to enable the agency to ascertain which 
records are being requested." To be sufficiently specific, a request must include identifying 
information about the subject matter, scope, and timeframe of the request. See Pennsylvania 
Department of Education v. Pittsburgh Post-Gazette, 119 A.3d 1121 (Pa. Cmwlth. 2015). Your 
request lacks sufficient scope and timeframe information. 

11. Your request is granted. The following record is being provided on the enclosed CD: 
"November2018_variables_weights.xlsx" 
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12. Your request is granted. The following record is being provided on the enclosed CD: 
"November2018_variables_weights.xlsx" 

13 Your request is granted in part and denied in part. Section 704 (b)(1) of the RTKL provides that 
an agency may respond to a request by notifying the requestor that the record is available 
through publicly accessible electronic means. Records responsive to this request are publicly 
accessible at: 

https://www.alleEhenvcountv.us/Human-Services/News- 

Events/Accomplishments/Alleshenv-Family-ScreeninE-Tool.aspx 

https://www.alleghenvcountvanalvtics.us/index.php/2017/Q4/17/developing-predictive- 

risk-models-support-child-maltreatment-hotline-screening-decisions/ 

Your request for "any notes taken during meetings that discussed the use of the software" 
must be denied pursuant to RTKL Section 708(b)(12), which exempts "notes and working 
papers prepared by or for a public official or agency employee used solely for that official's or 
employee's own personal use, including telephone message slips, routing slips and other 
materials that do not have an official purpose." 

14. Your request is granted in part and denied in part. Section 704 (b)(1) of the RTKL provides that 
an agency may respond to a request by notifying the requestor that the record is available 
through publicly accessible electronic means. Records responsive to this request are publicly 
accessible at: 

https://www.alleghenvcountvanalvtics.us/wp-content/uploads/2019/04/16-ACDHS- 
26 Ethical Package 102518.pdf#page=48 

https://www.alleghenvcountvanalvtics.us/wp-content/uploads/2019/04/16-ACDHS- 
26 Ethical Package 102518.pdf#page=57 

Additionally, the following records are being provided on the enclosed CD: 

"FAMILY SCREENING TOOL POLICY - 8-3-2016.pdf" 

"Family Screening Score Changes ll-30-2018_Redacted.pdf" 

This record has been redacted pursuant to RTKL Section 708(b)(28), which exempts records 
and information pertaining to individuals' applications for and receipt of social services, RTKL 
Section 708(b)(17), which exempts records relating to a noncriminal investigation, and RTKL 
Section 708(b)(30), which exempts records identifying the name, home address or date of 
birth of a child 17 years of age or younger. 

"Pred Analytics Field Screening Training_Redacted.pdf" 

This record has been redacted pursuant to RTKL Section 708(b)(28), which exempts records 
and information pertaining to individuals' applications for and receipt of social services, RTKL 
Section 708(b)(17), which exempts records relating to a noncriminal investigation, and RTKL 
Section 708(b)(30), which exempts records identifying the name, home address or date of 
birth of a child 17 years of age or younger. 
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"Pred Analytics Training - Fall 2016_Redacted" 

This record has been redacted pursuant to RTKL Section 708(b)(28), which exempts records 
and information pertaining to individuals' applications for and receipt of social services, RTKL 
Section 708(b)(17), which exempts records relating to a noncriminal investigation, and RTKL 
Section 708(b)(30), which exempts records identifying the name, home address or date of 
birth of a child 17 years of age or younger. 

15. Your request is granted. Please refer to the attachments and hyperlinks provided in response 
to requests 1 and 5 for records pertaining to requests for proposals. 

Additionally, the following records are attached: 

"Grantee Report 2018.pdf" 

"Grantee Report 2019.pdf" 

16. Your request is granted in part and denied in part. Section 704(b)(1) of the RTKL provides that 
an agency may respond to a request by notifying the requestor that the record is available 
through publicly accessible electronic means. Records responsive to this request are publicly 
accessible at: 

https://www.alleghenvcountvanalytics.us/wp-content/uploads/2019/04/16-ACDHS- 
26 Ethical Package 102518.pdf#page=48 

https://www.alleghenvcountvanalvtics.us/wp-content/uploads/2019/04/16-ACDHS- 
26 Ethical Package 102518.pdf#page=57 

Your request for a "validation review study" must be respectfully denied pursuant to RTKL 
Section 708(b)(9), which exempts "the draft of a bill, resolution, regulation, statement of 
policy, management directive, ordinance or amendment thereto prepared by or for an 
agency." 

17. Your request must be respectfully denied. Allegheny County cannot ascertain how to interpret 
your request for "the five most recent sets of data that were used for input, as well as the five 
most recent outputs of the software." But if a sufficiently specific timeframe were provided 
for this request, the request would be denied pursuant to RTKL Section 708(b)(28), which 
exempts records pertaining to individuals' applications for and receipt of social services, RTKL 
Section 708(b)(6)(i)(A), which exempts "a record containing all or part of a person's Social 
Security number; driver's license number; personal financial information; home, cellular or 
personal telephone numbers; personal email addresses; employee number or other 
confidential personal identification number, and RTKL Section 708(b)(30), which exempts "a 
record identifying the name, home address or date of birth of a child 17 years of age or 
younger." 

18. Your request must be respectfully denied. No records responsive to this request are in the 
possession of Allegheny County. 
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Sincerely, 



Jerry Tyskiewicz, Director 

County of Allegheny Open Records Officer 



RTK #4725 
Beryl Lipton 
MuckRock 

To Whom It May Concern: 

Pursuant to the Pennsylvania Right to Know Act, I hereby request the following records: 

• The initial 15 responses to the Request for Proposal issued by the Allegheny County Department of Human 
Services (DHS) to design and implement a system of decision-support tools and predictive analytics for 
human services, also known as the Allegheny Family Screening Tool 

• All review reports, notes, evaluations, and associated materials created by the RFP evaluation committee. 

• All deliverables provided by the winning applicants: Rhema Vaithianathan (AUT) and researchers from 
Auckland University of Technology (AUT), University of Southern California (USC), University of California- 
Berkeley and University of Auckland. 

• All minutes for meetings held to solicit community input on the creation of the tools, including the six 
project-specific meetings, any held at early stages of the project to collect feedback from key external 
stakeholders and funders, and the three open community meetings where over 30 stakeholder groups 
(including the Courts and the ACLU) were invited to discuss the work to date, implementation timeline and 
results. Please include all notes on the work to date, implementation timeline, and results. Please also 
include all project updates DHS shared with existing community networks including the Children's Cabinet 
and the Children, Youth and Families Advisory Board, and through the DHS Speaker Series. 

• All responses to the RFP for two independent evaluations of the AFST (process and impact) that was issued 
in 2015 and for which Hornby Zeller Associates was selected to conduct an process evaluation and Stanford 
University was selected to conduct an impact evaluation. 

• The process evaluation and the impact evaluation focused on whether the AFST increases the accuracy of 
decisions, reduces unwarranted variation in decision-making and reduces disparities, and overall referral 
rates and workload. 

• A list of sources/datasets - including child protective services, publicly funded mental health and drug and 
alcohol services, and bookings in the County jail - used by the AFST. 

• The current percentage of incoming referrals were not generating an AFST score. 

• The number of overrides conducted on "mandatory screen-ins" 

• Any correspondence and contract documents relating to the development and implementation of the 
program. 

• Records that show how the County's Integrated Data System is used in the program, including records that 
identify how risk factors are generated and the weights that are given to various risk factors. 

• Any software and algorithms developed for the implementation of the program. I'm particularly interested 
in any algorithms that have a public policy outcome, such as providing guidance on a policy or an assessment 
of an individual. More generally, I would like any software that was developed by or with, given to, used by, 
purchased or licensed to this agency for implementation of the above referenced program. This would 
include original source code, any compiled binaries (also known as executables), spreadsheets, program 
scripts, and other digital materials used to calculate any data in the above program. It would also include the 
input training data for machine learning algorithms. Collectively, these responsive materials will be referred 
to as "the software" in the following paragraphs. 



• All instructional materials, presentations and presentation materials (including recorded video and audio, 
PowerPoint files, prepared remarks, and slides formats), and other guidance on the use of "the software." 

This includes any notes taking during meetings that discussed the use of the software, any explanations 
(whether internally or externally generated) of how the software works, and any other document that has 
been used to help explain the use of "the software" to any party, including internal documentation, public 
relations materials, and executive summaries. This also includes any description of input and output data 
fields that will aid in understanding the type of information that is submitted to the software, and that is 
produced by the software. 

• A copy of any privacy impact assessments, use policies, standard operating procedures, data retention 
policies, legal opinions, warranties, non-disclosure agreements, contracts, liability waivers, insurance 
agreements, Requests for Proposals, Responses to Requests for Proposal, Memorandums of Understanding, 
Letters of Interest, usage policies, or informal agreements that reference "the software" or otherwise guide it 
use or the relationship between this agency and the provider or developer of the software. 

• A copy of any funding opportunity announcements, requests for proposals, grant applications and grantor 
status/progress reports, reports to legislative bodies, annual reports that mention the use of the Software, as 
well as audit records, including but not limited to security audits of the software, misuse reports, and reports 
to oversight bodies. 

• A copy of any validation or ethical review studies conducted with respect to the program or with respect to 
any software or algorithms used in connection with the program, and a copy of any records pertaining to 
inquiries for the need of validation studies or ethical reviews, or discussion of potential or actual validation 
studies or ethical reviews. By "validation study," I mean any study designed to assess how accurate the 
program is in predicting what it is supposed to predict, or to assess whether the program may err in the 
direction of either under- or overestimating likelihoods or predicted outcomes, or may produce any results 
that are biased or unexpected in any way. By "ethical review," I mean a review of how the design and/or 
implementation of the predictive algorithm conforms with ethical standards of fairness and privacy. 

• For the aspects of the software that require an input, for example, to compute a value, we request a copy 
of the five most recent sets of data that were used for input, as well as the five most recent outputs of the 
software, in whatever their native format is. If these inputs or outputs include exempt information, we ask 
that you exempt only that information which is specifically exempted from law, and provide the other 
material. 

If any of the records requested are protected by a nondisclosure agreement, I request that you provide 
documentation of that agreement. 

The requested documents will be made available to the general public, and this request is not being made for 
commercial purposes. 

In the event that there are fees, I would be grateful if you would inform me of the total charges in advance of 
fulfilling my request. I would prefer the request filled electronically, by e-mail attachment if available or CD- 
ROM if not. 

Thank you in advance for your anticipated cooperation in this matter. I look forward to receiving your 
response to this request within 5 business days, as the statute requires. 

Sincerely, 

Beryl Lipton 

Upload documents directly: https://www.muckrock.com/accounts/agency_login/allegheny-county- 
department-human-services-17470/allegheny-family-screening-tool-rfp-responses-and-development- 
materials-allegheny-county-department-of-human-services-70873/?uuid-login=3fd23b4b-91a6-44bl-a691- 
04ab2bc66bf8#agency-reply 



